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To  ensure  that  the  Department  of  Defense  (DoD)  is  approving  Milestone  B  decisions 
for  programs  that  are  technologically  mature,  the  director  of  Defense  Research  and 
Engineering  (DDR&E)  implemented  Technology  Readiness  Assessments1  (TR A)  as 
a  DoD  5000  requirement.  With  the  continued  and  growing  dependence  of  major  DoD 
systems  on  information  technologies,  the  TRA  Deskbook  [1]  is  being  updated  to 
address  the  unique  aspects  of  computer  systems,  hardware,  and  software  technologies  for 
embedded  systems,  business  management  information  systems,  net-reliant  systems  (eg., 
command  and  control),  and  infrastructure  systems  (eg.,  net-centric  enterprise  services). 

The  DDR&E  tasked  the  Institute  for  Defense  Analysis,  assisted  by  representatives 
from  DoD  services  and  agencies  to  develop  the  revised  guidelines.  This  article  summa¬ 
rises  the  revised  guidelines,  including  the  new  Software  Technology  Readiness  levels,  and 
provides  examples  in  applying  the  updated  guidelines  to  major  defense  acquisitions. 


Technology  Readiness  Assessments1 
(TRAs)  are  conducted  on  major 
defense  acquisitions,  which  often  consist 
of  complicated  machinery  and  hardware 
systems  that  depend  on  advances  in  state- 
of-the-practice  technologies.  The  intent  of 
a  TRA  is  to  document  that,  prior  to  system 
design  and  development,  there  is  a  reason¬ 
able  expectation  that  the  acquisition  is 
technically  feasible.  In  other  words,  die 
effort  being  undertaken  is  likely  to  be  real¬ 
ized  with  currently  available  technologies. 
The  TRAs  focus  is  technologies  —  it  is  not 
intended  to  address  the  capabilities  of  die 
acquiring  or  developing  organizations,  nor 
does  it  attempt  to  assess  processes  being 
applied  during  development. 

The  TRA  is  mandated  by  Department 
of  Defense  (DoD)  Directive  5000.1  and 
DoD  Instruction  5000.2.  The  TRA 
Deskbook  [1],  approved  by  the  deputy 
undersecretary  of  defense  for  Science  and 
Technology  (DUSD[S&T)),  describes  the 
TRA  requirements  and  process  in  detail.  It 
has  recentiy  been  revised  to  address  some 
of  the  unique  needs  of  information  tech¬ 
nology  (IT)-based  systems.  Completion  of 
die  TRA  allows  early  identification  of  tech¬ 
nology  issues  so  they  can  be  addressed  as  an 
integral  part  of  die  development  process. 
Potentially  costiy  changes  in  the  later  stages 
of  system  development,  where  even  small 
modifications  can  be  costiy  and  time  con¬ 
suming,  can  be  mitigated  or  avoided. 

The  current  TRA  process  follows  diree 
basic  steps:  identification  of  critical  tech¬ 
nology  elements  (CTEs),  evaluation  of 
CTE  maturity  using  technology  readiness 
levels  (TRLs),  and  maturation  planning. 
The  program  manager  and  die  Component 
Science  and  Technology  executive  are 
joindy  responsible  for  determining  die  final 


list  of  CTEs,  assessing  its  maturity,  and 
finalizing  any  necessary  maturation  plans. 
The  DUSD(S&T)  is  responsible  for  over¬ 
sight  of  the  TRA  process  and  providing  a 
yearly  summary  report  to  Congress. 

Motivation  for  the  Revised 
TRA  Deskbook 

The  current  TRA/TRL  model  works  well 
for  traditional  hardware-oriented  systems 
being  managed  to  a  set  of  capabilities  and 
requirements  documents  with  few  interde¬ 
pendencies  with  other  systems.  However, 
an  increasing  number  of  defense  acquisi¬ 
tions  are  either  information  systems  or  tra¬ 
ditional  systems  widi  increasing  dependen¬ 
cies  on  computer  technologies.  Those  diat 
are  not  classified  in  die  acquisition  system 
as  information  systems  directly  may  have  a 
large  IT  component  or  a  large  dependency 
upon  success  of  the  IT  component.  To 
address  diis  fundamental  change  in  die 
types  of  acquisitions,  a  corresponding 
change  in  the  approach  to  TRAs  was  need¬ 
ed.  This  keeps  TRAs  relevant  to  DoD’s 
changing  acquisition  needs  while  providing 
die  same  level  of  technology  analysis  and 
management  associated  widi  traditional 
hardware  systems. 

The  problem  faced  with  information 
systems  is  that  very  few  hardware  and  soft¬ 
ware  elements  can  be  singled  out  as  CTEs. 
As  a  result,  die  TRA  skips  over  many 
important  issues  that  lie  outside  of  hard¬ 
ware  and  software.  These  can  collectively 
be  termed  IT  issues  and  include  interfaces, 
diroughput,  scalability,  external  dependen¬ 
cies,  and  information  assurance.  These  are 
integral  to  how  the  system  is  designed.  The 
use  of  these  technologies  is  critically 
dependent  upon  a  system  architecture  diat 


drives  system  interdependencies  and  com¬ 
plexities.  The  system  architecture  defines 
which  of  these  issues  are  important  to 
consider  and  which  may  have  associated 
CTEs  beyond  those  directly  related  to  sys¬ 
tem  functionality. 

The  Nature  of  IT  Systems 

IT  systems  fall  into  four  basic  types  that 
the  DoD  procures.  While  many  interme¬ 
diates,  flavors,  and  special  cases  exist, 
characterizing  DoD  acquisition  into  these 
four  pipes  allows  us  to  more  readily  ensure 
that  our  revised  approach  to  TRA  is  effec¬ 
tive  and  provides  useful  advice  on  con¬ 
ducting  the  TRA.  Following  are  the  four 
IT  system  types: 

•  Business  systems. 

•  Net-reliant  (batde  management)  sys¬ 
tems. 

•  Network  infrastructure  (or  services 
provider). 

•  Embedded  systems. 

While  each  of  these  systems  has  many 
points  of  overlap,  they  also  have  unique 
requirements;  we  will  briefly  discuss  each 
one.  Some  acquisitions  may  include  a 
combination  of  the  above,  so  the  TRA 
may  include  characteristics  of  several  of 
these  types. 

Business  Systems 

Business  systems  acquisitions  typically 
consist  of  a  small  set  of  large  commercial 
off-the-shelf  (COTS)  products  to  which 
the  organization  will  adapt.  The  business 
system  may  be  characterized  by  using  off- 
the-shelf  information  system  components 
and  COTS  software  together  in  a  new 
environment  to  support  the  business  and 
management  functions  of  an  organization. 
Typical  business  systems  include  finan- 
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cial  management,  personnel  management, 
and  enterprise  resource  planning.  The  ITs 
are  die  primary  CTEs  with  their  configura¬ 
tion  driving  additional  CTE  designation. 
Typically,  the  CTEs  will  align  with  the 
COTS  products  selected.  Additional  CTEs 
may  come  in  the  form  of  case  legacy  con¬ 
version  tools,  and  environments  may  be 
critical  to  keep  backward  compatibility  and 
seamless  data  access. 

Net-Reliant  Systems 

Net-reliant  systems  provide  military 
(warfighting)  functions  that  rely  on  data 
exchanges  with  physically  disparate  ele¬ 
ments.  These  systems  involve  large 
amounts  of  data  push  (control)  or  data 
pull  (awareness)  function  and  are  typically 
command  and  control;  battle  management 
systems;  or  intelligence,  surveillance,  and 
reconnaissance  systems.  The  net-reliant 
system  is  characterized  by  an  intense  real¬ 
time  requirement,  a  heavy  reliance  on 
exchanges  with  external  information 
sources  or  consumers,  and  may  be  pushing 
die  state-of-the-art  in  data  fusion  and 
blackboard  collaboration. 

The  emphasis  in  these  systems  is  hav¬ 
ing  computers  assist  humans  in  awareness 
and  decision-making  processes  across 
physically  separate  warfighting  and  sensing 
elements.  The  software  run-time  environ¬ 
ment  for  real-time  applications  may  be 
critical  here  as  the  functionality  may  be 
safety-  or  security-related. 

The  ability  to  keep  communication 
lines  open  is  likely  to  lead  to  a  number  of 
unique  CTEs  by  itself.  The  architecture 
will  include  strong  information  assurance 
requirements.  Reach-back  support  and 
voyeur  channels  will  most  likely  identify 
critical  elements  from  the  information 
assurance  perspective.  CTEs  may  also 
enable  efforts  to  manage  data,  translate 
data,  and  establish  composability  (how  sys¬ 
tems  bind  to  one  ano titer).  The  IT  that 
realizes  tite  system  and  the  elements  cited 
above  should  be  considered  for  CTEs. 

Network  Infrastructure 

Network  infrastructure  system  acquisi¬ 
tions  provide  the  equipment  and  capabili¬ 
ties  necessary  for  the  successful  operation 
of  net-reliant  systems.  Backbone  and 
Services  systems  acquisition  technology 
issues  often  manifest  themselves  as  tite 
maturity  of  standards  (often,  but  not 
always  commercial)  and  standardization 
that  transcends  individual  COTS  or  gov¬ 
ernment  off-the-shelf  (GOTS)  products. 
Timeliness  and  robustness  of  services  are 
major  technology  considerations  that 
should  be  included  when  assessing  maturi¬ 
ty.  The  network  infrastructure  is  character¬ 


ized  by  large  database  management  and 
glue  logic  to  execute  and  retrieve  services 
across  a  Wide  Area  Network  of  varying 
security.  This  environment  is  critical  and 
unique  and  die  IT  elements  are  most  cer¬ 
tainly  a  CTE.  Since  COTS  has  not  operat¬ 
ed  in  this  environment  before,  anything  of 
a  critical  nature  must  be  demonstrated,  and 
die  separation  of  security  streams  must  be 
considered  as  a  CTE. 

Embedded  Systems 

Embedded  warfighting  systems  such  as  a 
tank,  ship,  or  aircraft  are  systems  whose 
functions  are  focused  on  warfighting  plat¬ 
forms,  and  whose  functionality  is  enabled 
by  IT  but  not  driven  by  IT  itself. 
Embedded  systems  emphasize  using  com¬ 
puter  hardware  and  software  to  automate 
internal  functions  of  a  weapon  system 
such  as  platform  control  and  status,  sensor 
signal  and  data  processing,  and  weapons 
tasking.  The  embedded  systems  range 
from  simple  to  complex  and  emphasize 
autonomous  functionality  in  timeframes 
meaningful  to  a  computer. 

Embedded  system  acquisitions  may 
include  full  development  (where  the  infor¬ 
mation  technology  is  a  primary  issue)  to 
modification  of  existing  systems  (informa¬ 
tion  architecture  is  firm,  and  demonstrated 
in  an  operational  environment)  where 
information  technology  is  not  an  issue. 
The  environments  that  convert  software  to 
firmware  may  be  CTEs.  Real  time  is  often 
critical  —  making  die  timing  associated  with 
any  calculation  routine  a  part  of  die  CTE 
determination  consideration.  Few  oppor¬ 
tunities  exist  to  use  COTS  or  GOTS 
beyond  microprocessors  and  operating 
systems  because  these  systems  are  largely 
unprecedented. 

Summary  of  Changes  to  the 
TRA  Deskbook 

To  address  the  unique  aspects  of  IT  and 
IT-based  systems,  the  DUSD(S&T)  has 
developed  a  set  of  software  TRLs  (see 
Table  1)  and  has  provided  additional  guid¬ 
ance  and  examples  on  how  environmental 
issues  unique  to  IT  systems  should  be 
addressed  in  IT  system  TRAs. 

CTE  Determination 

CTE  determination  for  IT  and  IT-based 
system  TRAs  must  begin  with  die  basic 
expectations  (requirements,  capabilities, 
functions)  for  the  acquisition.  The  TRA 
includes  a  mapping  of  CTEs  to  those 
expectations.  For  IT  and  IT-based  systems 
particularly,  expectations  may  not  be  driv¬ 
en  from  a  top-down  set  of  anticipated 
functionality. 


Some  IT  system  acquisitions  include 
technology  modernization  issues  driven  by 
supportability  and  compatibility  that  could 
provide  a  source  of  nontraditional  CTEs 
such  as  online  software  configuration 
management  and  update  technologies. 
Other  IT  systems  acquisitions  include 
modernization  as  a  way  to  realize  transfor¬ 
mational  concepts.  Integration  and  roll-out 
efforts  may  also  be  technology-enabled 
with  new  or  novel  technologies  enabling 
those  parts  of  the  acquisition  as  well. 

The  new  suite  of  IT-unique  CTEs 
requires  a  different  line  of  thinking  when 
marketplace  considerations,  technology 
trends,  and  the  short  shelf  life  of  IT  tech¬ 
nologies  are  viewed  in  die  context  of  long- 
lived  DoD  acquisition  programs.  CTE 
considerations  in  these  situations  might 
transcend  an  individual  product,  but  may 
consider  the  capabilities  provided  by  a  sta¬ 
ble  set  of  suppliers  and  customers  as  a 
whole.  For  example,  middleware  products 
supported  by  consortium-facilitated  stan¬ 
dards  might  provide  die  necessary  technol¬ 
ogy  stability  while  die  actual  suite  of  avail¬ 
able  products  may  change  from  year  to 
year.  Careful  consideration  is  needed  when 
selecting  a  particular  technology  that  is 
vendor-specific  and  not  widely  embraced 
across  both  the  vendor  base  and  industry- 
and  government-standard  bodies. 

Environments  and  Information 
Architecture 

The  above  considerations  imply  that,  at 
Milestone  (MS)  B  of  die  DoD  5000  series, 
there  is  some  form  of  notional  architec¬ 
ture  for  the  system  acquisition.  Whether  it 
is  a  high-level  diagram  of  interrelation¬ 
ships  between  COTS  products  or  a  set  of 
data  flow  diagrams  for  developed  software 
elements,  the  existence  of  system  architec¬ 
ture  must  be  available.  Architectural  con¬ 
siderations  are  present  in  the  consideration 
of  environments  in  die  analysis  of  CTE 
maturity.  A  COTS  CTE  may  be  mature  in 
that  it  has  been  used  by  several  large 
organizations  outside  die  DoD  for  similar 
purposes,  but  the  DoD’s  unique  architec¬ 
ture  renders  much  of  that  maturity  uncer¬ 
tain  because  of  differences  in  information 
assurance,  data  management,  etc.  To  reach 
the  upper  levels  of  maturity  (TRLs  6  and 
higher),  a  successful  operation  in  a  similar 
or  identical  environment  to  that  anticipat¬ 
ed  for  die  acquisition  is  necessary.  The 
maturity  of  the  definition  of  die  system 
architecture  itself  and  how  CTEs  are  inte¬ 
grated  and  demonstrated  as  a  result  of  this 
will  impact  die  maturity  assessment  of  a 
particular  CTE. 

IT  systems  have  the  additional  com¬ 
plexity  that  architectures  and  architectural 
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TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles 
observed  and 
reported. 

Lowest  level  of  software  technology  readiness;  a  new  software 
domain  is  being  investigated  by  the  basic  research  community. 
This  level  extends  to  the  development  of  basic  use,  basic 
properties  of  software  architecture,  mathematical  formulations, 
and  general  algorithms. 

Basic  research  activities,  research  articles, 
peer-reviewed  white  papers,  point  papers,  and 
early  lab  model  of  basic  concept  may  be  useful 
for  substantiating  the  TRL  level. 

2 

Technology  concept 
and/or  application 
formulated. 

Once  basic  principles  are  observed,  practical  applications  can 
be  invented.  Applications  speculative  and  there  may  be  no 
proof  or  detailed  analysis  to  support  the  assumptions. 

Examples  are  limited  to  analytic  studies  using  synthetic  data. 

Applied  research  activities  analytic  studies, 
small  code  units,  and  papers  comparing 
competing  technologies. 

3 

Analytical  and 
experimental  critical 
function  and/or 
characteristic  proof 
of  concept 

Active  research  and  development  is  initiated.  The  level  at 
which  scientific  feasibility  is  demonstrated  through  analytical 
and  laboratory  studies.  This  level  extends  to  the  development 
of  limited  functionality  environments  to  validate  critical 
properties  and  analytical  predictions  using  non-integrated 
software  components  and  partially  representative  data. 

Algorithms  run  on  a  surrogate  processor  in  a 
laboratory  environment,  instrumented 
components  operating  in  laboratory 
environment,  and  laboratory  results  showing 
validation  of  critical  properties. 

4 

Module  and/or 
subsystem  validation 
in  a  laboratory 
environment,  i,e., 
software  prototype 
development 
environment. 

Basic  software  components  are  integrated  to  establish  that 
they  will  work  together.  They  are  relatively  primitive  with 
regard  to  efficiency  and  robustness  compared  with  the 
eventual  system,  Architecture  development  initiated  to  include 
interoperability,  reliability,  maintainability,  extensibility, 
scalability,  and  security  issues.  Emulation  with  current/  legacy 
elements  as  appropriate.  Prototypes  developed  to 
demonstrate  different  aspects  of  eventual  system. 

Advanced  technology  development,  standalone 
prototype  solving  a  synthetic  full-scale  problem, 
or  standalone  prototype  processing  fully 
representative  data  sets. 

5 

Module  and/or 
subsystem  validation 
in  a  relevant 
environment. 

Level  at  which  software  technology  is  ready  to  start  integration 
with  existing  systems.  The  prototype  implementations  conform 
to  target  environment/interfaces.  Experiments  with  realistic 
problems.  Simulated  interfaces  to  existing  systems.  System 
software  architecture  established  Algorithms  run  on  a 
processor(s)  with  characteristics  expected  in  the  operational 
environment. 

System  architecture  diagram  around 
technology  element  with  critical  performance 
requirements  defined,  processor  selection 
analysis,  and  Sim/Stim  laboratory  buildup  plan. 
Software  placed  under  configuration 
management.  COTS/GOTS  in  the  system 
software  architecture  are  identified. 

6 

Module  and/or 
subsystem  validation 
in  a  relevant  end-to- 
end  environment. 

Level  at  which  the  engineering  feasibility  of  a  software 
technology  is  demonstrated.  This  level  extends  to  laboratory 
prototype  implementations  on  full-scale  realistic  problems  in 
which  the  software  technology  is  partially  integrated  with 
existing  hardware/software  systems 

Results  from  laboratory  testing  of  a  prototype 
package  that  is  near  the  desired  configuration 
in  terms  of  performance,  including  physical, 
logical,  and  data  and  security  interfaces. 
Comparisons  to  tested  environment  to 
operational  environment  analytically 
understood.  Analysis  and  test  measurements 
quantifying  contribution  to  system-wide 
requirements  such  as  throughput,  scalability, 
and  reliability.  Analysis  of  human-computer 
(user  environment)  begun. 

7 

System  prototype 
demonstration  in  an 
operational  high 
fidelity  environment, 

Level  at  which  the  program  feasibility  of  a  software  technology 
is  demonstrated.  This  level  extends  to  operational  environment 
prototype  implementations  where  critical  technical  risk 
functionality  is  available  for  demonstration  and  test  in  which 
the  software  technology  is  well  integrated  with  operational 
hardware/software  systems. 

Critical  technological  properties  are  measured 
against  requirements  in  a  simulated  operational 
environment. 

8 

Actual  system 
completed  and 
mission  qualified 
through  test  and 
demonstration  in  an 
operational 
environment. 

Level  at  which  a  software  technology  is  fully  integrated  with 
operational  hardware  and  software  systems.  Software 
development  documentation  is  complete.  All  functionality 
tested  in  simulated  and  operational  scenarios. 

Published  documentation,  product  technology 
refresh  build  schedule,  and  software  resource 
reserve  measured  and  tracked 

9 

Actual  system 
proven  through 
successful  mission- 
proven  operational 
capabilities. 

Level  at  which  a  software  technology  is  readily  repeatable  and 
reusable.  The  software  based  on  the  technology  is  fully 
integrated  with  operational  hardware/software  systems.  All 
software  documentation  verified.  Successful  operational 
experience.  Sustaining  software  engineering  support  in  place. 
Actual  system. 

Production  configuration  management  reports, 
technology  integrated  into  a  reuse  wizard,  and 
out-year  funding  established  for  support 
activity. 

Table  1:  Revised  Software  TRJ_j 
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issues  transcend  any  single  CTE.  In  these 
cases,  additional  mitigation  plans  may  be 
warranted  when  technology  issues  are 
revealed  as  a  result  of  environmental  con¬ 
siderations,  or  die  architectures  are  defined 
after  MS  B  as  a  part  of  the  development 
effort. 

Technology  Maturity  and 
Demonstrations 

The  current  requirement  for  major 
defense  acquisition  programs  at  MS  B  is 
that  all  CTEs  be  maturity  level  TRL  6  or 
higher,  or  have  a  maturation  plan  to 
achieve  TRL  6  or  higher  when  needed. 
Achieving  TRL  levels  of  6  or  higher 
depends  upon  CTEs  successfully  running 
in  a  relevant  or  operational  environment. 

Prior  to  MS  B,  activities  such  as  con¬ 
cept  development  and  experimentation 
should  include  a  significant  amount  of 
prototyping  or  pilot  demonstrations.  It  is 
important  that  these  demonstration  efforts 
collect  the  necessary  information  to 
inform  future  acquisitions  regarding  die 
successes  and  weaknesses  of  a  vendor 
product  or  a  particular  implementation  so 
that  die  program  development  and  sup¬ 
ported  capability  expectations  are  known. 
In  some  cases,  a  concept  demonstration 
may  use  a  development  environment  that 
will  require  upgrading  for  production. 

Laboratory  and  pilot  demonstrations 
are  likely  to  examine  the  interconnections, 
database  manipulations,  and  preliminary 
data  on  throughput,  execution,  and 
resource  utilization.  External  dependen¬ 
cies  for  specific  technologies  and  technol¬ 
ogy  insight  should  be  identified  in  detailed 
Technology  Transition  Agreements 
(TTAs)  between  the  supplying  organiza¬ 
tion  and  the  receiving  acquisition  program 
office  (more  information  on  TTAs  can  be 
found  in  die  TRA  Deskbook).  Protocols 
needed  to  resolve  the  external  dependen¬ 
cies  are  worked  out.  If  external  dependen¬ 
cies  involve  another  program  office’s 
development,  dien  schedules  are  synchro¬ 
nized,  and  risk  abatement  activities  are 
undertaken  diat  may  include  alternative 
elements  and  key  action  dates  (for  execut¬ 
ing  alternate  plans)  tied  to  the  external 
program’s  ability  to  demonstrate  its  tech¬ 
nology  readiness. 

It  is  during  diese  early  prototype  and 
demonstration  activities  that  tradeoffs  are 
often  made  as  to  what  elements  of  soft¬ 
ware  reside  in  each  part  of  the  architec¬ 
ture,  distributed  versus  centralized  data 
and  control,  information  assurance 
aspects,  and  preliminary  data  on  through¬ 
put  and  network  requirements.  It  is  here 
diat  most  of  the  hardware  and  software 
elements  are  identified.  At  its  conclusion, 
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we  have  a  detailed  architecture  and  a  defi¬ 
nition  of  the  totality  of  the  relevant  envi¬ 
ronment.  This,  in  turn,  allows  for  sys¬ 
tem/  subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 
that  is  needed  for  TRL  6  and  higher.  This 
detailed  architecture  is  die  work  break¬ 
down  structure  for  critical  technology 
assessment.  In  many  instances  the  actual 
elements  may  be  off-the-shelf  and  not  a 
technology  issue,  but  their  integration  and 
the  architecture  are  technology  issues. 
Often,  the  glue  logic  that  allows  die  various 
elements  of  the  system  to  work  together 
will  be  a  critical  technology. 


“Because  business 
management  systems 
will  largely  consist  of 
COTS  products ,  the  TRA 
should  begin  with  an 
analysis  of  the  maturity 
of  the  chosen  products. ” 


Advice  forTRAs  on  the 
Various  Types  of  IT  Systems 
Business  Systems 

Because  business  management  systems 
will  largely  consist  of  COTS  products,  die 
TRA  should  begin  with  an  analysis  of  die 
maturity  of  die  chosen  products.  Critical 
COTS  products  will  be  those  that  provide 
mission-essential  functionality  but  are 
either  new  or  novel  by  organizational 
experience,  or  because  of  the  environment 
in  which  they  will  be  running  have  ele¬ 
ments  on  which  die  COTS  products  have 
not  been  used. 

Typically,  these  products  have  been 
used  in  non -DoD  organizations  of  com¬ 
mensurate  size  so  a  high  degree  of  maturi¬ 
ty  is  expected.  However,  die  DoD’s  execu¬ 
tion  environment  has  a  number  of  unique 
aspects  diat  prevent  us  from  assuming  diat 
success  outside  the  DoD  will  automatical¬ 
ly  imply  success  for  us,  including  informa¬ 
tion  assurance,  technologies  for  handling 
classified  data,  unique  legacy  applications, 
net-centricity,  data  management  mecha¬ 
nisms,  number  of  users,  etc. 

The  TRA  should  include  not  only  die 
CTE  maturity  but  also  a  detailed  analysis 
of  environmental  issues  that  could  impact 
die  ability  of  die  COTS  products  to  suc¬ 
cessfully  execute.  Finally,  die  environmen¬ 
tal  assessment  should  also  include  the  abil¬ 


ity  of  the  collection  of  products  to  suc¬ 
cessfully  run  together. 

As  an  example,  a  systems  center  wants 
to  change  its  financial  management  and 
accounting  system  to  a  suite  of  commer¬ 
cial  products  used  by  many  major  corpora¬ 
tions.  The  center  did  a  series  of  pilots  widi 
the  anticipated  COTS  applications  on  their 
existing  hardware  and  operating  environ¬ 
ment  to  understand  bodi  die  impacts  to 
the  users  as  well  as  die  viability  of  die 
applications  in  die  unclassified  and  classi¬ 
fied  systems  on  which  financial  and 
accounting  data  is  stored  and  managed. 
Upon  reasonably  successful  conclusion  of 
the  pilot  programs,  die  systems  center 
decides  to  proceed  widi  die  conversion 
with  some  minor  modifications  of  die 
chosen  suite  of  applications. 

For  the  TRA,  consideration  of  die 
CTEs  begins  widi  a  listing  of  die  COTS 
products  being  used  in  the  project  along 
with  any  external  technologies  such  as  die 
existing  desktops  and  servers  on  which 
these  applications  will  run,  upon  which 
success  of  die  effort  depends.  Several  of 
the  minor  applications  might  fall  off  the  list 
because  these  programs  are  not  critical  to 
successful  functioning  of  the  system  or 
because  diere  are  many  other  applications 
that  could  reasonably  take  dieir  place.  The 
existing  suite  of  desktop  computers,  net¬ 
works,  and  servers  would  not  make  the  list 
of  CTEs  eidier  because  diat  IT  has  been 
successfully  operating.  The  proposed  list 
of  CTEs  consists  of  the  small  list  of  appli¬ 
cations  that  are  both  critical  to  success  and 
are  new  or  novel  in  the  sense  that  diey 
have  not  run  with  DoD  information  assur¬ 
ance  technologies,  DoD  legacy  applica¬ 
tions,  and  in  the  DoD’s  data  environment. 

The  proposed  list  of  CTEs  is  reviewed 
and  die  final  list  is  analyzed  to  determine 
TRL  ratings  based  upon  industry  experi¬ 
ence  and  pilot  results.  Where  a  CTE  was 
not  piloted  and  has  not  been  previously 
used  by  the  DoD  in  a  similar  environment, 
it  can  achieve  a  rating  of  no  higher  than 
TRL  5.  A  piloted  CTE  can  typically 
achieve  a  rating  of  TRL  6  or  7,  assuming 
no  major  problem  was  encountered  during 
the  pilot  effort.  The  TRA  should  also 
include  a  careful  analysis  of  the  environ¬ 
ment  and  multi-application  issues  that 
might  not  have  been  considered  when 
viewing  the  applications  by  diemselves. 

Issues  such  as  timeliness  of  system- 
wide  transactions,  impact  of  information 
assurance  policies,  and  the  presence  of 
multiple  sources  of  data  on  the  system 
should  be  considered  along  widi  the  need 
for  diese  applications  to  peacefully  coexist 
on  die  computing  system  as  part  of  die 
environmental  analysis.  Where  a  CTE  is 
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less  than  TRL  6,  or  an  environmental  issue 
raises  significant  concerns,  a  maturation 
plan  should  be  developed  and  any  corre¬ 
sponding  risk  item  entered  into  die  pro¬ 
gram  manager’s  risk  management  system. 

Net-Reliant  Systems 

For  net-reliant  systems,  the  TRA  process 
should  start  with  the  set  of  proposed 
capabilities  and  examine  die  criticality  of 
die  technologies’  associated  capabilities. 
Because  the  solution  is  likely  to  consist  of 
a  mix  of  COTS,  GOTS,  and  developed 
software,  the  TRA  will  likely  encompass  all 
elements  of  an  IT  system.  These  technolo¬ 
gies  should  be  viewed  in  die  context  of 
their  ability  to  move  and  manage  data  with 
external  systems  so  the  environmental 
considerations  may  have  unique  technolo¬ 
gy  aspects  by  themselves.  Some  of  the 
ability  to  move  data  will  depend  upon 
capabilities  provided  by  the  Net  Infra¬ 
structure  systems.  As  a  minimum,  these 
dependencies  should  be  identified  and 
briefly  analyzed.  Where  the  infrastructure 
piece  is  immature,  or  presents  a  new  envi¬ 
ronment  for  COTS/GOTS,  the  analysis 
for  die  specific  COTS/ GOTS  will  need  to 
be  considered. 

An  acquisition,  in  this  case,  might  be  a 
command/ control  or  sensor  net  that  man¬ 
ages  and  assimilates  data  from  a  variety  of 
physically  separated  and  disparate  plat¬ 
forms.  This  type  of  system  might  include 
an  improved  version  of  several  GOTS 
products  enabled  by  a  small  suite  of  COTS 
technologies  combined  with  upgrades  to 
existing  data  communications  networks  to 
achieve  an  end-to-end  capability  across 
existing  warfighting  platforms. 

The  CTE  list  will  focus  on  die  COTS 
and  GOTS  products  that  are  new  or  novel 
as  supported  by  previous  experience  and 
demonstrations.  The  CTE  list  will  also 
include  any  cross  dependencies  with  other 
acquisition  efforts  such  as  upgrades  to  die 
communications  equipment  to  support  the 
net-reliant  system.  Where  such  a  depend¬ 
ency  exists  between  military  programs,  die 
TTA  plays  a  critical  role  to  detail  the  inter¬ 
dependencies  between  the  acquisition, 
resource  sponsor,  science  and  technology 
activity,  and  odier  project  managers  to 
develop,  deliver,  and  integrate  a  technolo¬ 
gy  or  product  into  the  acquisition. 

Much  like  other  systems,  higher  levels 
of  maturity  are  achieved  by  pilot  or  opera¬ 
tional  experience  in  an  environment  diat 
closely  resembles  die  one  anticipated  for 
final  operation.  The  central  processing  and 
display  suite  may  or  may  not  be  a  CTE 
depending  upon  die  amount  of  opera¬ 
tional  experience  widi  die  anticipated  set 
of  technologies.  Where  a  CTE  is  deter¬ 


mined  to  be  a  maturity  less  dian  TRA  6, 
die  appropriate  maturation  plan  and  risk 
items  should  be  established  and  noted  in 
die  TRA. 

Network  Infrastructure 

Network  infrastructure  TRAs  need  to  con¬ 
sider  the  maturity  of  both  the  technology 
standards  under  which  die  netted-elements 
will  be  interoperable  as  well  as  die  maturi¬ 
ty  of  technologies  associated  with  unique¬ 
ly  acquired  elements  of  the  infrastructure 
acquisition  (data  services,  networking, 
etc.).  CTEs  should  be  drawn  from  bodi 
sources  and  consider  the  interaction  and 
compatibility  of  both  sets  of  technologies. 
Because  these  acquisitions  will  become  an 
integral  part  of  the  run-time  environment 
for  other  systems,  diere  could  be  addition¬ 
al  technology  considerations  for  the  ability 
to  roll  these  technologies  out  to  the  other 
diree  types  of  IT  systems.  TRA  should  not 
stray  into  analysis  of  die  roll-out  process 
unless  that  process  is  enabled  by  a  specific 
technology  such  as  automated,  net- 
enabled  configuration  management  tools. 

An  example  in  this  area  might  be  a 
new  network  combining  land  lines,  radio 
frequency  links,  and  a  suite  of  servers  for 
data  and  application  hosting  to  support  a 
combination  of  business  and  warfighting 
systems.  Here  the  CTE  analysis  will  start 
with  a  set  of  technologies  taken  from  the 
anticipated  system  architecture  and  con¬ 
sider  any  new  or  novel  aspects  to  the 
applications  such  as  data  rates,  operational 
hardening  (harsh  environments,  jam 
resistance,  DoD-unique  encryption),  and 
authentication  or  collaboration  services 
diat  will  impact  the  ability  of  die  pro¬ 
posed  suite  of  technologies  to  provide  the 
required  capabilities. 

As  with  net-reliant  systems,  there  will 
likely  be  a  combination  of  COTS  and 
GOTS  products  combined  with  some 
operational  experience  of  a  smaller  scale. 
In  this  situation,  the  acquisition  may  not 
depend  upon  other  acquisitions  to  fulfill 
its  requirements  but  may  be  die  dependen¬ 
cy  for  several  other  acquisitions.  The  net- 
infrastructure  acquisition  may  be  signatory 
on  multiple  TTAs  as  a  supplier  of  a  critical 
technology  for  other  systems.  Where  such 
TTAs  exist,  they  should,  as  a  minimum,  be 
noted  in  the  TRA.  As  before,  achieving 
ratings  of  TRL  6  or  higher  depends  upon 
pilot  or  operational  experience. 

Embedded  Systems 

Embedded  systems  are  largely  DoD- 
unique  warfighting  capabilities  diat  are 
enabled  by  IT  systems  rather  than  driven 
by  IT  systems.  While  improvements  in  IT 
capabilities  (memory  density,  power  con¬ 


sumption,  processing  speed)  are  critical  to 
warfighting  systems,  most  of  the  function¬ 
ality  is  not  commercially  available,  resulting 
in  large  amounts  of  developed  software  or 
software  reused  from  previous  embedded 
systems  acquisitions.  IT  technologies  are 
not  generally  CTEs  in  and  of  themselves 
except  where  the  unique  requirements  of 
the  embedded  system  (such  as  radiation 
hardening)  result  in  the  development  or 
use  of  military-unique  technologies.  With 
our  growing  dependence  on  software,  die 
sheer  scale  of  software  development  or 
integration  may  be  considered  a  CTE 
where  enabled  by  a  particular  set  of  tech¬ 
nologies  diat  are  new  or  novel. 

For  example,  assume  the  acquisition  is  a 
complex  warfighting  platform  such  as  an 
aircraft,  ground  vehicle,  or  ship.  In  these 
types  of  acquisitions,  die  capabilities  of  the 
platform  are  enabled  by  computers,  but  die 
computers  themselves  are  not  new  or 
novel.  Here,  the  CTE  analysis  begins  with 
an  architecture  of  die  major  warfighting 
functions  (rather  than  specific  hardware  or 
software  elements),  or  subsystems  where 
domain  maturity  exists.  The  analysis  in  this 
case  might  show  that  neither  die  functions 
nor  the  realization  of  those  functions  in 
hardware  and  software  is  new  or  novel. 
However,  the  anticipated  design  of  die 
platform  includes  consolidation  of  all  die 
major  computer-based  functions  on  a 
reconfigurable  suite  of  computing 
resources  (processors,  memory,  and  display 
stations)  networked  across  die  platform. 

Here,  die  TRA  for  the  IT  elements  of 
the  platform  will  focus  on  the  ability  of 
available  technology  to  support  this  recon¬ 
figurable  suite  of  computing  resources  and 
run  all  the  anticipated  applications  in  a 
safe,  reliable,  timely  manner  as  document¬ 
ed  in  the  design  scenarios.  This  situation 
will  likely  result  in  a  single,  complex  TRA 
for  the  computing  suite  diat  should  be 
supported  by  significant  piloting  and  pro¬ 
totyping  prior  to  MS  B  to  achieve  ratings 
greater  than  TRL  5. 

Note  diat  with  the  complexities  of 
today’s  acquisitions,  a  system  may  contain 
elements  of  two  or  more  of  the  system 
types  noted  in  the  preceding  sections.  In 
these  cases,  as  well  as  any  of  those  falling 
into  a  single  bin,  the  need  for  experienced, 
professional  judgment  is  critical.  A  TRA  is 
not  a  substitute  for  a  project  manager  and 
acquisition  center  staff  who  are  technical¬ 
ly  qualified  and  actively  involved  in  an 
acquisition.  The  TRA  provides  a  focal 
point  and  emphasis  on  technology  issues 
such  as  major  milestone  reviews,  and  is  a 
voice  for  die  efforts  of  die  technical  staff 
and  their  contributions.  One  common 
theme  that  occurs  in  all  these  types  of 
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acquisitions  is  the  need  for  prototyping 
and  demonstrations  to  provide  a  sound 
basis  from  which  to  proceed. 

Summary 

In  an  effort  to  assure  that  major  acquisi¬ 
tion  programs  adequately  address  technol¬ 
ogy  issues,  die  DoD  requires  a  TRA.  The 
TRA  should  assist  program  offices  in  die 
evaluation  and  maturation  of  technology 
issues  present  in  DoD  acquisitions  as  well 
as  provide  the  approving  acquisition  offi¬ 
cial  assurance  that  the  program  under 
review  has  a  sound  technology  foundation. 
In  today’s  information  systems,  the  TRA 
includes  both  the  hardware  and  software 
of  the  system  as  they  fit  within  die  archi¬ 
tecture  used  to  integrate  these  elements 
and  the  intended  environment  in  which 
diey  will  run.  IT  is  becoming  a  critical  fac¬ 
tor  in  die  success  of  modern  DoD  infor¬ 
mation  systems.  The  TRA  provides  a 
chance  to  analyze  a  program  and  assess  die 


technological  maturity  of  its  hardware, 
software,  and  IT.  For  technologies  of 
insufficient  maturity,  a  program  of 
demonstrations  and  prototypes  should  be 
established  to  provide  a  mature  set  of  ITs 
that  are  ready  to  support  system  develop¬ 
ment  when  needed. ♦ 
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Note 
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series,  the  Acquisition  Guidebook,  and 
the  TRA  Deskbook  for  the  latest 
approved  policy  and  guidance. 
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